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Preface 


Stackable credential programs—that is, programs that facilitate students’ ability to earn 


multiple postsecondary certificates or degrees—have been a priority for Ohio, which 
has a long history of legislation, state and regional initiatives, and institution-led efforts 
to build more-effective pathways to address the needs of employers and students. To 
assess progress and inform ongoing efforts to scale stackable credentials, the Ohio 
Department of Higher Education and the RAND Corporation established a research 
partnership and began work on the Ohio Stackable Credentials Project. 


In May 2020, RAND released Stacking Educational Credentials in Ohio: Pathways 
Through Postsecondary Education in Health Care, Manufacturing and Engineering 
Technology, and Information Technology,' a report that documented how students were 
moving through stackable credential pipelines in Ohio postsecondary institutions be- 
tween 2005 and 2015 and noted some potential areas for improvement in the pipeline. 
This toolkit is intended to be a companion document to that report, providing a how-to 
guide for practitioners who are overseeing stackable credential programs to identify 
issues in their stackable credential pipelines and make data-driven improvements. The 
toolkit describes a four-step process that practitioners can use to (1) take high-level 
statewide findings and collect additional evidence to explore these issues in their con- 
text, (2) focus on specific levers for change, (3) identify improvements and a framework 
for assessing them, and (4) carry out rapid cycles of improvement. This four-step pro- 
cess was adapted from a prior RAND toolkit on postsecondary continuous improvement, 
Tools for Improving Corequisite Models: A Guide for College Practitioners.” Three of the 
findings from the May 2020 RAND report on stacking of credentials in Ohio are used— 
i.e., lower rates of stacking for black students and adult learners, lower rates of stacking 
for students who earn a certificate at an Ohio Technical Center, and excess credit hours 
among students who stack credentials—to illustrate how this four-step process can be 
applied to improve stackable credential pipelines. 


The study was undertaken by RAND Education and Labor, a division of the RAND Corpo- 
ration that conducts research on early childhood through postsecondary education pro- 
grams, workforce development, and programs and policies affecting workers, entrepre- 
neurship, and financial literacy and decisionmaking. This study was sponsored primarily 
by the ECMC Foundation and was also supported by funding from the U.S. Department 
of Education’s Institute of Education Sciences (grant number R305H190033). 


' Lindsay Daugherty, Jenna W. Kramer, Drew M. Anderson, and Robert Bozick, Stacking Educational Creden- 
tials in Ohio: Pathways Through Postsecondary Education in Health Care, Manufacturing and Engineering Technolo- 
gy, and Information Technology, Santa Monica, Calif: RAND Corporation, RR-A136-1, 2020. 

> Lindsay Daugherty, Rita Karam, Daniel Basco, and Julia H. Kaufman, Tools for Improving Corequisite Models: 
A Guide for College Practitioners, Santa Monica, Calif.: RAND Corporation, TL-319-IES, 2019. 
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The opinions expressed in this toolkit are the authors’ alone and do not represent 
the views of the ECMC Foundation or the Institute of Education Sciences. 


More information about RAND can be found at www.rand.org. Questions about 
this toolkit should be directed to Lindsay Daugherty at Idaugher@rand.org, and 
questions about RAND Education and Labor should be directed to 
educationandlabor@rand.org. 
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An Overview of the Toolkit 
Who Is the Toolkit for? 


This toolkit will be useful to individuals overseeing stackable credential programs at post- 
secondary education institutions who are interested in using data-driven approaches to 


improve how students move through these programs. 


What Is the Purpose of the Toolkit? 


In a recent RAND Corporation report, Stacking Educational Credentials in Ohio: Path- 
ways Through Postsecondary Education in Health Care, Manufacturing and Engineering 
Technology, and Information Technology, a group of researchers documented the com- 
pletion of postsecondary certificate and degree programs in health care, manufacturing 
and engineering technology (MET), and information technology (IT) between 2005 

and 2015.! The researchers identified some possible areas for improvement, such as 
improving rates of stacking for black students and adult learners and reducing excess 
credit hours. This toolkit aims to provide a clear process by which institutions can take 
the high-level findings from the report and use them to drive on-the-ground improve- 
ment to stackable credential pipelines. And although the examples in this toolkit focus 
on stackable credential pipelines, these tools and processes can be used to improve a 
variety of different programs and policies in postsecondary institutions. 


How Is the Toolkit Organized? 


The toolkit contains the following sections. 


A Four-Step Process for Addressing Issues in the Stackable 
Credential Pipeline 

The toolkit first provides an overview of a basic process and a set of tools for using data 
to drive improvement. 


« Step 1. Assess evidence at your own institution(s) to identify issues in the 
stackable credential pipeline to tackle. This section provides some tips on differ- 
ent types of data that can be examined to identify issues in the stackable credential 
pipeline. 


Step 2. Focus on the root causes to identify specific areas for improvement. 
This section introduces a cause-and-effect diagram—or fishbone diagram—a tool 
for getting stakeholder input to determine the root causes of issues that postsec- 
ondary institutions face. 


Step 3. Determine what improvements will be tested and how progress will be 
measured. This section describes the Model for Improvement, a framework that 
institutions can use to guide their improvement work. 
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« Step 4. Test, monitor, and refine improvements through an iterative pro- 
cess. This section provides an overview of how practitioners can carry outa 
Plan-Do-Study-Act (PDSA) cycle. 


Application of the Four-Step Process to Issues in the 
Stackable Credential Pipeline 

The four-step process is then applied to three challenges in stackable credential 
pipelines that were identified based on findings from RAND’s Stacking Education- 
al Credentials in Ohio report. The toolkit illustrates how to use the four-step pro- 
cess to address the following three challenges, drawing on fictional descriptions 
of how an institution might carry out improvement efforts: 


¢ Challenge 1. Increasing opportunities for stacking among black 
certificate-earners: Although black students were well represented among stu- 
dents who completed certificate programs in Ohio, the researchers found that 
these students were less likely to go on and stack additional credentials after 
completing the certificate when compared with white certificate-earners. In this 
example, an institution tests out advising changes to address concerns about 
equity in student awareness of stackable credential programs. 


« Challenge 2. Increasing opportunities for stacking among students who 
earn a certificate at an Ohio Technical Center (OTC): OTCs provide many 
of the certificates earned in the state of Ohio, but the researchers found that 
somewhat fewer of these students went on to earn additional credentials 
relative to students who earned certificates from community colleges and 
universities. In this example, a regional consortium of institutions considers 
improvements to encourage more OTC certificate-earners to return for addi- 
tional credentials. 


Challenge 3. Reducing excess credit hours among students who stack 
credentials: When researchers compared students who earned a certifi- 

cate and an associate’s degree with those who earned only the associate’s 
degree, the researchers found that those who earned both a certificate and 
an associate’s degree were earning an average of 17 additional credits. These 
“excess credit hours” were particularly high for health care students. These 
inefficiencies in credit accumulation can be costly to students in terms of both 
time and money and can also stunt their progress through the program. In this 
example, an institution tests out streamlined programs as a strategy to reduce 
excess credit accumulation. 


Because all of our three example sections apply the same process and have similar 
content, it might be sufficient to read through only one or two of them to understand 
the process rather than reading the toolkit from cover to cover. 
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Final Considerations for Using the Four-Step Process 

The toolkit concludes with a brief overview of some common barriers to practitioner- 
driven improvement efforts and a discussion of strategies that can be used to overcome 
these barriers. 


Supplementary Worksheets 
Blank worksheets are provided for printing and use by toolkit users as they carry out 
the four-step process.” 


How Was This Toolkit Created? 


In 2019, RAND released a toolkit titled Tools for Improving Corequisite Models: A Guide 
for College Practitioners.* This toolkit focused on a popular reform in postsecondary 
education—corequisite remediation—and offered a how-to guide for practitioners in 
postsecondary institutions who were interested in conducting rapid-cycle improvement 
and evaluation strategies to improve their corequisite approaches. Development of the 
four-step process for this toolkit drew heavily from the content of that original toolkit 
that focused on quality improvement approaches. For example, both toolkits use similar 
content to describe how to conduct root cause analysis, how to develop a model for 
improvement, and how to carry out a PDSA cycle.* As noted earlier, this toolkit also 
draws from a RAND report, Stacking Educational Credentials in Ohio: Pathways Through 
Postsecondary Education in Health Care, Manufacturing and Engineering Technology, 
and Information Technology. Findings were pulled directly from that report to structure 
the three examples that are used to apply the four-step process in this toolkit. The con- 
tent for this toolkit was refined through input from policymakers at the Ohio Department 
of Education, practitioners at Ohio postsecondary institutions, and peer reviewers. 
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A Four-Step Process for 
Addressing Issues in the 
Stackable Credential Pipeline 


Many states and postsecondary institutions are focused on building programs 
and systems in which students have opportunities to earn (or “stack”) multiple 
credentials as they progress through a career, a concept referred to in this toolkit 
as stackable credentials. Stackable credential programs are considered valuable 
for students because they offer multiple on-ramps and off-ramps to education and 
careers, providing an alternative to students who might need to pursue nontra- 
ditional routes through education as they balance the demands of college with 
work and family obligations.* These programs also aim to map credentials directly 
onto specific career opportunities, providing individuals with a clear pipeline for 
upskilling throughout their careers.° But stackable credentials are complex to 
build, and there are likely to be areas where institutions can improve opportuni- 
ties for students to stack credentials. But how do institutions figure out where to 
focus their improvement efforts? And how should institutions go about making 
improvements? 


This section describes a simple four-step process through which practitioners at 
postsecondary institutions can make improvements to their stackable credential 
programs. In these four steps, practitioners will learn to (1) assess evidence at their 
own institutions to identify issues in the stackable credential pipeline to tackle; 

(2) focus on the root causes of those issues to identify specific areas for improve- 
ment; (3) determine what improvements will be tested and how progress will be 
measured; and (4) test, monitor, and refine improvements through an iterative pro- 
cess. The process draws on common approaches to continuous improvement and 
evaluation. This overview of the process is followed by three examples of how the 
process might be applied to issues in the stackable credential pipeline. 


Step 1: Assess Evidence in Your Own Context 
to Identify Issues in the Stackable Credential 


Pipeline to Tackle 


The RAND report on the stacking of credentials in Ohio between 2005 and 2015 
identified several issues in stackable credential pipelines, such as racial dispar- 
ities in rates of stacking and excess credit hours earned among students with 
multiple credentials.’ But do those issues identified in historical data still 
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exist, and are they relevant at your institution? Institutions also may face other issues 
with stackable credential pipelines, such as problems with transferring credits across 
institutions, excess credits earned by students who stack credentials, and misalignment 
between program offerings and employer needs. You will likely want to examine some 
evidence from your own context before deciding on a specific area on which you would 
like to focus your improvement efforts. You might also be alerted to potential issues 

in stackable credential pipelines in other ways, such as feedback from faculty. In this 
case, you might also need to examine additional data to determine whether this issue is 
pervasive across classrooms, campuses, and/or programs. 


The first step is to gather some additional evidence to identify and describe the issue you 
would like to tackle. But where do you start? What types of data should you gather? This 
section provides some tips. Your institutional research department might also be able to 
provide some assistance on issues around data collection. 


Tip 1: Map out a set of questions that you would like to answer to better under- 
stand the issues. Before you collect and/or analyze data on stackable credential 
programs and practices, it is always a good idea to map out the specific set of questions 
you would like to answer. This will help you ensure that data collection is focused, and 
you can quickly focus in on an area of the stackable credential pipeline where improve- 
ment can be made. For example, some questions you might ask include the following: 


Do patterns in the data from our institution look similar to patterns in other data 
that suggest issues with stackable credential pipelines (e.g., statewide historical 
data, reports on other institutions)? 


How do programs, policies, and practices within the institution or broader educa- 
tional system contribute to the issue? 


To what degree are issues identified in one program or classroom present across 
classrooms, programs, student subgroups, and/or campuses across the institution? 


What do key stakeholders (e.g., faculty, advisers, students) see as the biggest 
barriers that might prevent students from stacking credentials? Where can improve- 
ments be made to better support students and/or staff? 


Where does the institution have leverage to make changes to programs, policies, 
and/or practices within the institution or broader system? What are the pros and 
cons of making adjustments? 


Tip 2: Choose the data sources that are appropriate for answering your questions. 
Different types of data are appropriate for answering different types of questions about 
the stackable credential pipeline. For example, if your question calls for examining 
patterns of student course-taking and credential completion, you will usually want to 
rely on administrative data and calculate rates of stacking for your institution using 
National Student Clearinghouse data. If you would like to collect information from many 
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students or staff members on the challenges they are facing, their perspectives 
on programs or services, or outcomes that are not covered in administrative data, 
a survey might be more appropriate. If you want to more deeply understand the 
experiences of individuals participating in or providing stackable credential pro- 
grams, you might want to consider focus groups, interviews, or informal conversa- 
tions. Observations can also provide rich data on what is happening in classrooms 
and how individuals within the institution interact. Program documentation and 
data can also provide valuable evidence on how different parts of the stackable 
credential pipeline are working. Table 1 describes how these different types of 
data sources are best used, as well as some of their limitations. 


Step 2: Focus on the Root Causes to Identify 
Specific Areas for Improvement 


After identifying the area in the stackable credential pipeline where you would 
like to make improvement, it is important to narrow in and focus on the root 
causes of an issue. For example, disparities in the rates of credential stacking 
among adult learners and younger students might be driven by the advising 
provided by the institution, the structure of programs, student decisionmaking, 
or other factors. Rather than focusing improvement efforts on the broader goal 
of “addressing lower rates of credential stacking among adult learners,” it is 
better to focus in on what might be driving those lower rates of stacking—i.e., 
the areas where changes can be made by the institution to drive improvement. 


There are many strategies and tools that you can use to determine what is under- 
lying your issues in the stackable credential pipeline. One common approach is a 
structured team process called root cause analysis, which uses a cause-and-effect 
diagram—or fishbone diagram—to map out the root causes of the issue that you 
are facing.* An example of a fishbone diagram is presented in Figure 1, and a blank 
worksheet is included in the appendix. The process for completing a fishbone dia- 
gram includes the following steps: 


1. Assemble a team of stakeholders representing the full range of individuals 
who play a role in the stackable credential pipeline. These individuals might 
include administrators, faculty, other school staff, students, and external 
stakeholders. 


2. Come to a consensus regarding the stackable credential pipeline issue that 
you would like to address, and write that issue at the mouth of the “fish.” In 
Figure 1, the issue at the mouth of the fish is “lower rates of stacking among 
adult learners.” 


3. With the full team’s input, define the different types of drivers that might 
cause that problem (e.g., people, policies). The example in Figure 1 contains 
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Table 1. Different Sources of Evidence That Might Shed Light on Issues in the Stackable 


Credential Pipeline 


Data Source 


Best Used For 


Limitations 


Administrative data 


Providing systematic data across broad 
groups of students or staff 

Measuring academic success outcomes for 
students 

Examining variation across students while 
accounting for differences in student, class- 
room, and instructor characteristics 
Limiting data collection burden on students 
or staff members 


Limited to the measures already being collect- 
ed for other purposes; can be hard to add new 
information 

Might take time to process and be ready for use 
Might not include information on all types of cre- 
dentials (e.g., certifications) 


Surveys 


Providing systematic data across broad 
groups of students or staff members 


Providing data on how a program is being 
rolled out, perspectives of individuals, suc- 
cesses and challenges, and explanations 
for trends in quantitative data 


Assessing outcomes not captured in admin- 
istrative data 


Can take a moderate amount of time and resourc- 
es if creating new surveys 

Can suffer from low response rates that limit rep- 
resentativeness 

Accuracy of self-reported data depends on who 


responds and how much effort they provide when 
answering questions 


Focus groups/ inter- 
views 


Providing in-depth data from a moderate 
number of individuals 


Providing data on how a program is being 
rolled out, perspectives of individuals, suc- 
cesses and challenges, and explanations 
for trends in quantitative data 


(Focus groups) Providing a setting where 
respondents can interact and build from 
one another’s ideas 


(Interviews) Providing individuals with 
privacy to discuss sensitive issues and 
perspectives 


Might cover few individuals, and might not be 
representative 


Accuracy of self-reported data can be affected by 
response bias and respondent engagement and 
interactions 

Accuracy of self-reported data depends on who 
participates and how engaged they are in answer- 
ing questions 


Observations of 
classrooms or inter- 
actions 


Program documents 
or data (e.g., course 
documents, sign-in 
sheets, time logs, 
adviser case man- 
agement notes) 


Providing detailed data on how a program 
is being rolled out 

Assessing interactions between various 
groups 

Measuring the requirements for a course or 
program 

Understanding how a program is intended 
to be implemented 


Collecting data with limited burden on 
students or school staff participants 


Might cover few individuals, and might not be 
representative 

Accuracy of data depends on the ability to collect 
consistent data across multiple observers 
Information might vary across classrooms, pro- 
grams, and campuses 

If not already collected for program operation, 
data can be time-consuming to collect 


Usefulness and accuracy of data depend on the 
level of effort and consistency in data collection 
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Figure 1. Example of a Fishbone Diagram Describing the Issue of Disparities in Credential 


Stacking Among Adult Learners 


Factors outside college Advising 


Jobs, families 
limit availability 


More work experience 
and better job options 
without stacking 


Students not talking 


with advisers Limited awareness 
of options to stack 


Information on the 


website not clear 


Issue 
Lower rates 
of stacking 
among adult 
learners 


Limited math 
Higher course 


Choice of programs that readiness 
do not stack well failure rate 
Lack of time to Lack of night/ 
Barriers to transferring use tutoring weekend course 
credits offerings 
Program/institution Courses/instruction 


NOTE: Primary drivers are written in plain text, while secondary drivers are written in italics. 


four types of drivers: factors outside of college, advising, program or institu- 
tional drivers, and course or instructional drivers. 


. Break individuals into subgroups to focus in on the different categories iden- 


tified in Step 3 and identify the root causes in this area that might be driving 
your issue of focus. Ask, “Why does this happen?” to identify different caus- 
es of the problem, and record these on the “bones” of the fish. These initial 
sets of root causes can also be referred to as primary drivers. In Figure 1, 
examples of primary drivers include limited math ability, limited awareness 
of options to stack, and jobs or family responsibilities that limit availability. 


. Now, try to think about the root causes underlying these primary drivers 


identified in Step 4, breaking the issue down until participants cannot think of 
any additional drivers of the issue. These are often referred to as secondary 
drivers. For example, in Figure 1, unclear information on the website and a lack 
of interaction between advisers and students are shown as secondary drivers 
that may be contributing to limited awareness of opportunities to stack. 
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6. Come back together as a group to report back and collectively assess the different 
drivers that you have identified (e.g., through ranking) to determine (1) which root 
causes stakeholders believe are most critical to driving the issue and (2) where the 
institution has leverage to drive improvement. 

In all, the process might take approximately 90 minutes. Root cause analysis typically 
incorporates the voices of the full range of individuals who have roles or perspectives 
on the issue with the stackable credential pipeline, so you will want to assemble a larg- 
er group to weigh in. As you engage in the fishbone diagram exercise, you might need 
to consider strategies for gathering the input of this broad group, such as breaking in- 
dividuals out into smaller groups, using sticky notes to allow individuals to suggest root 
causes, and asking participants to vote for the top areas in which they believe improve- 
ments should be made. Several resources are available that provide tips on carrying out 
root cause analysis, for those interested in additional guidance.’ 


Ideally, you will want to narrow down the focus of the improvement efforts to just one 
of these root causes. This focus on a single narrow root cause allows you and your 
improvement efforts to stay focused and increases the chance that concrete change can 
be made in that one area over a relatively short period of time. The root cause on which 
improvement efforts are focused can also be called the problem of practice. In the 
example provided in Figure 1, a problem of practice might be improving the quality and 
consistency of information and guidance that students receive on stackable program 
opportunities as part of their certificate programs to ensure that students are aware of 
their options. Another problem of practice might be enhancing night and evening pro- 
gram offerings to ensure that working adults have access to the courses they need. 


Step 3: Determine What Improvements Will Be Tested 
and How Progress Will Be Measured 


Before you begin rapid cycles of improvement around your problem of practice, it is 
important to start with a clear roadmap for what changes will be tested and how im- 
provement will be measured. This is referred to as the model for improvement. Defining 
a model for improvement requires you to address the following three questions: 


1. Aim: What are you trying to accomplish? 

2. Measures: How will you know that a change is an improvement? 

3. Changes: What changes will you make that will result in improvement?!° 
The aim is similar to your problem of practice, though you must set a more specific 
goal for what you would like to achieve with regard to the problem of practice. Aim 
statements often include explicit numbers and timelines. For example, if the problem 
of practice focused on a need to increase access to night and weekend course offerings, 
the aim might be “to increase adult learner enrollment in night and weekend courses 
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by 50 percent within the next two academic years.” This aim statement provides 
your improvement team with a clear marker of when you have made the necessary 
improvements and can move onto addressing other problems of practice. 


While the aim is your long-term goal for the improvement cycle and typically 
requires you to measure at least one key outcome, you will also need to develop 
a broader set of measures that can be used to assess progress on a shorter time 
frame as you are testing out different changes to your stackable credential pipe- 
lines. Continuous improvement experts often call for new changes to be tested 

in cycles that last 90 days or less, so it would be ideal to focus on early indicators 
of improvement—things that can be assessed within a single semester. Measures 
may focus on processes, outputs, perceptions of individuals, or outcomes. In addi- 
tion, you will want to include measures that are tied to the secondary drivers that 
your improvement efforts seek to change, as well as the primary drivers and the 
larger issue on which you aim to see improvement. For example, if you are testing 
out various changes to expand enrollment in night and weekend courses, you 
might want to look at measures that include the total number of night and week- 
end sections that were scheduled, the proportion of sections that were filled, the 
perceptions of adult learners as to whether course offerings were sufficient to 
accommodate the schedules, and overall adult learner enrollment with the new 
mix of offerings. 


Finally, you will need to identify one or more changes that you would like to 

test out in your PDSA cycles (Step 4). There are several different ways you might 
identify changes that might lead to improvement. First, it can be valuable to look 
to the research and talk with other institutions to identify promising practices that 
have been tested in other contexts. Second, the evidence you collected in Step 1 
and the root cause analysis in Step 2 might have provided a deeper understand- 
ing of the problem of practice and how it might be best addressed. Finally, it is 
important to talk with staff who are involved in the delivery of stackable credential 
programs (and possibly with students) to gather their input on changes that could 
be made to achieve the desired improvements. 


A blank worksheet is provided in the appendix for documenting your model for 
improvement. 
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Step 4: Test, Monitor, and Refine Improvements 
Through an Iterative Process 


Now that you have identified your issue in the stackable credential pipeline, narrowed 
your focus to a specific problem of practice that had contributed to driving that issue, 
and identified one or more changes that you would like to make, it is time to begin test- 
ing out those changes. One common process that can be used to test out those chang- 
es is the PDSA approach (Figure 2), which calls for a series of rapid, iterative cycles of 
testing and monitoring.'! For each PDSA cycle, you will ideally settle on just one change 
that will be tested. The following subsections provide a brief overview of the key activi- 
ties that take place in each of the four stages of the PDSA cycle. 


Key Activities in the Plan Stage 


The Plan stage is about developing a roadmap for what will be done throughout the 
PDSA cycle. There are four types of activities that commonly take place in the Plan 
stage: (1) identifying questions to be addressed in the cycle and making predictions, 

(2) determining which data will be used to assess improvement, (3) laying out the activ- 
ities that will take place throughout the cycle and developing a timeline, and (4) assign- 
ing roles and responsibilities. 


As part of the model for improvement that you developed in Step 3, you had to identify 
measures that could be used to assess progress on your problem of practice. These 
measures can be used to identify questions about the change that is being tested and 
make predictions about what you anticipate you might find. For example, if one of your 
measures for assessing expanded night and weekend courses is the degree to which 
scheduled sections are being adequately filled, you might ask, “What percentage of 
night and evening sections were canceled because of low enrollment?” You will then 
want to make a specific prediction, such as “fewer than 5 percent of scheduled night 
and evening sections will be canceled because of insufficient enrollment.” Similar to 
measures, inquiry questions and predictions can focus on how a program is rolled out, 
the perceptions of individuals, and/or outcomes. A worksheet for documenting your 
model for improvement is included in the appendix. 


Next, you will need to determine which data will be used to answer your inquiry 
questions. The goal is not to conduct a full-scale evaluation; the questions you identify 
to assess improvement and the data needed to answer those questions should be rela- 
tively easy for practitioners to collect in real time and analyze within a single semester. 
Although some examples of different types of data sources were provided in Table 1, 
the rapid nature of PDSA cycles might limit what can be collected and require creative 
thinking. For example, you might need to administer short three- to five-question “pulse 
surveys” of students and/or staff at the end of a course session or meeting to collect 
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Figure 2. Key Activities in a Plan-Do-Study-Act Cycle 
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quick feedback on progress. Data that are collected on program operations or 
data that exist on individuals in various information systems (e.g., early alert sys- 
tems, personnel records) might be useful. Notes from meetings and other informal 
data sources might also provide valuable information. A worksheet for document- 
ing your data collection and analysis plan is included in the appendix. 
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After making predictions and identifying which data are needed, you must lay out the 
activities for the other stages of the PDSA cycle and develop a timeline. As noted 
earlier, continuous improvement experts call for 90 days or less per PDSA cycle, though 
one-year cycles might be sufficiently ambitious if the improvement team is learning the 
process. For example, in a one-year cycle, an institution might plan for the cycle starting 
in the summer, pilot a change and collect data in the fall, study data in the winter, and 
take action and begin a new cycle in the spring. You will want to lay out the detailed set 
of activities required to roll out the change, as well as the activities required to collect 
data, study the change, consider broad perspectives, and take action. For example, 

if you are planning to test out an expanded set of night and weekend course offer- 

ings, your improvement team will need to figure out which programs and courses the 
expanded offerings will be available, find faculty to teach these courses, and provide 
information to students to get them into the courses. The PDSA cycle will also require 
the improvement team to collect data on the measures of improvement. To keep these 
activities on track with a relatively short timeline, it will be essential to establish key 
deadlines and deliverables throughout the cycle. A worksheet for documenting your 
activities, timeline, and roles and responsibilities is included in the appendix. 


Once you have identified the key activities that will take place in the cycle, it is criti- 

cal for the improvement team members and develop a timeline to assign roles and 
responsibilities for overseeing a change and evaluation activities. Your core improve- 
ment team should consist of individuals who will oversee the process throughout (and, 
ideally, who were involved in the root cause analysis and developing your model for 
improvement in Steps 2 and 3) and should include individuals who have deep knowl- 
edge of the credential pipeline and the specific issue being tackled through continuous 
improvement. Continuous improvement teams can also benefit from including someone 
familiar with data collection and analysis, whether that be an internal staff member from 
institutional research or an external consultant. Assigning clear responsibilities and 
deliverables to this team for core activities is essential to staying on track. As you plan, 
you might also want to consider how to engage others outside the group who will need 
to contribute in various ways to ensure that the improvement activities are carried out 
successfully. You might need to engage with various administrators, faculty, advisers, 
students, employers, or other staff throughout a PDSA cycle. 


Key Activities in the Do Stage 


The Do stage is about delivering the program or initiative and collecting data to under- 
stand whether these efforts are leading to improvement. This section describes two 
types of activities that commonly take place in the Do stage: (1) delivering the specific 
change that is being tested and (2) collecting data to inform improvement. 


MAKING IMPROVEMENTS TO STACKABLE CREDENTIAL PIPELINES 


Study @ 


First, you will need to deliver the specific change that is being tested to 

address the problem of practice. In an initial PDSA cycle, you might want to roll 
out a new change to just a few programs, one campus, or a small group of staff. 
The PDSA cycle will provide quick feedback on whether the change being tested 
is an improvement, and then you can scale and/or refine the change in future 
PDSA cycles (or abandon the change if it is not an improvement). Staff involved in 
delivering the program should have clear guidance and ongoing support to en- 
sure that they understand the expectations for how they should be delivering the 
change. For example, guidance and support might include documents describing 
improvement activities, a workshop or training that introduces those changes, and 
ongoing check-in calls to answer questions and offer help as needed. And even 
though the change is being tested on a smaller scale, it is important to ensure that 
the change is being delivered as it would be if scaled throughout the institution to 
ensure that the findings are relevant in both cases. 


The Do stage also requires you to collect data to inform improvement as the 
program or initiative is rolled out. Although you will want to continue to examine 
longer-term student success outcomes (e.g., persistence, stacking of credentials) 
to identify issues in the pipeline (Step 1), you will need to collect data that track 
progress within shorter periods (i.e., within a semester) to allow for quicker as- 
sessments of progress in rapid PDSA cycles. Depending on the specific problem of 
practice that you choose related to stackable credential pipelines, you might want 
to collect data on program offerings, course offerings and enrollments, advising 
processes and interactions with students, instructional processes and interactions 
with students, student engagement (e.g., attendance, homework completion), or 
student or staff satisfaction and perceptions of effectiveness. Some of these data 
might already be collected in your institution, but in other cases they might not. 
Collecting high-quality data might require leadership support and strong buy-in 
from the staff involved in collecting the data. 


Key Activities in the Study Stage 


The Study stage is the point at which you will take a look at the findings, reflect on 
them, and identify possible courses of action based on these results. This section 
describes three types of activities that commonly take place in the Study stage: 

(1) analyzing the data, (2) reflecting on findings with a broad group of stakehold- 
ers, and (3) documenting what was learned. 


Analyzing the data should be somewhat straightforward, given that the ques- 
tions and predictions developed during the Plan stage provide a road map for 
your analysis. In many cases, PDSA cycles might focus on analyzing data from 
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only the group testing out the new improvement, so your predictions might rely on 
before-and-after comparisons for the group piloting the change. Alternatively, you 
might establish benchmarks for measuring progress (e.g., “fewer than 10 percent of all 
night and weekend sections will be canceled because of insufficient enrollment”) that 
are set by practitioners at the college according to their expertise and view of what 
signifies improvement and success. Another way to set a benchmark for measuring 
progress is to use a comparison group. For example, you might compare night and 
weekend offerings and enrollments across programs that did and did not expand their 
night and weekend offerings. 


Once the core improvement team has had an opportunity to analyze the findings on the 
key questions for the PDSA cycle, you should share these findings with other individuals 
who might have important perspectives. Reflecting on findings with a broad group of 
stakeholders will help ensure that the improvement team did not overlook important 
perspectives on the findings that are evident to a broader set of individuals involved in 
program delivery. In addition, the opportunity to engage with the findings and pro- 

vide input might help build buy-in among the individuals who are essential to driving 
improvement. For example, you might want to share findings on the expansion of night 
and weekend classes with faculty teaching those courses, adult learners you aim to 
serve through those courses, or advisers scheduling students into those courses. 


Finally, you should make sure to clearly document what was learned from the PDSA 
cycle so that future cycles can build on what has been learned. These findings serve 
as a source of evidence for identifying other issues in the stackable credential pipe- 
line, can be shared with other programs and institutions as lessons learned, and can 
serve as an internal record to avoid duplication of changes that have failed to generate 
improvement in the past. 


Key Activities in the Act Stage 


Once you have identified the problem of practice, gathered some evidence to inform 
improvement, and reflected on those findings, the final stage of the PDSA cycle is to 
take action to improve the program or policy. The Act stage commonly includes three 
types of activities: (1) determining what actions should be taken, (2) sharing findings and 
improvement plans broadly, and (3) identifying questions that require additional study. 


You now have evidence of whether the change you tested led to improvements, and 
you must determine what actions should be taken. These actions might include 
abandoning the change, modifying the change, or scaling it as is. For example, if the 
evidence looks promising on the efforts to expand evening and weekend sections for 
two programs, you might decide to expand these changes to other programs in your in- 
stitution. At the same time, you might also need to continue to refine the changes, such 
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as tinkering with which courses are offered at these times. As you scale and refine 
your change, you can engage in additional PDSA cycles to continue monitoring 
and refining your improvement efforts. Alternatively, if your PDSA cycle does not 
yield evidence that the change led to improvement, it might be best to abandon 
the change and shift your focus to other changes you decided you wanted to test 
in Step 3 or other root causes you identified as requiring improvement in Step 2. 


To mobilize individuals to carry out these improvements and to anticipate chal- 
lenges that might arise, you should share findings and improvement plans with 
a broad group of individuals involved in carrying out these changes. 


In many cases, a PDSA cycle might raise as many questions as it provides an- 
swers, and you will want to identify issues that require further study. For 
example, you might find that few adult learners are aware of the new evening 
and weekend course offerings and that you now need to tackle the issue of 
getting this information out. Institutions typically engage in many PDSA cycles 
until a problem of practice has been sufficiently addressed. Once you have made 
sufficient improvement on one problem of practice, you can move on to another 
problem of practice. 
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Application of the Four- 
Step Process to Issues in the 
Stackable Credential Pipeline 


This section of the toolkit provides examples that demonstrate how the four-step pro- 


cess might be used to address challenges in stackable credential pipelines. 


Challenge 1: Lower Rates of Stacking Among Black 
Certificate-Earning Students 


A recent RAND report suggests that black certificate-earners in Ohio institutions were 
less likely to return and complete additional postsecondary credentials.!” This find- 

ing serves as the jumping-off point for a fictional example in which a postsecondary 
institution carries out the four-stage improvement process to make improvements to 
their stackable credential pipeline and address issues of racial equity in the stacking of 
credentials. 


Step 1: Assess Evidence in Your Own Context to Identify 
Issues in the Stackable Credential Pipeline to Tackle 

Using statewide data, RAND researchers found that 27 percent of black students who 
earned an initial certificate in health care, MET, or IT between 2005 and 2013 went on 
to earn an additional credential within two years, compared with 31 percent of white 
certificate-earners (Figure 3).'> The racial disparities in the rates of stacking were 
particularly large for MET and IT fields. This suggests that there might be issues in the 
stackable credential pipeline that are acting as barriers for black students in particular. 


You are leading a task force at your institution charged with improving stackable cre- 
dential pipelines in common technical fields. After looking at the statewide findings on 
racial disparities in stacking, you would like to consider this issue as a potential area for 
improvement in your institution. Before you dive into improvement efforts to address 
this issue, the first step is to gather some additional evidence to better understand the 
issue within your own context. 


Your team members work collaboratively to identify five questions they would like to 
answer to better understand the issues related to racial disparities in the stackable 
credential pipeline, as well as data sources that can be used to address these questions 
(Table 2). Your team has questions about whether the disparities exist at your institution 
(and in which programs), as well as what aspects of the pipeline might be driving these 
disparities. For example, you want to determine whether black certificate-earners at 
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Figure 3. Percentage of First-Time Certificate-Earners Who Stacked One or More Additional Cre- 
dentials Within Two Years, by Field and Race/Ethnicity 
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SOURCES: This figure was drawn from Daugherty et al., 2020. Data for the figure were drawn from the Higher 
Education Information System (HEI) and OTC data files in the Ohio Longitudinal Data Archive (Ohio Longitudinal 
Data Archive, data repository, Columbus, Ohio: Ohio State University, Center for Human Resource Research, 
undated). 

NOTE: The figure presents the percentage of individuals who earned an initial certificate in each of the three 
fields and overall (three fields combined) between 2005 and 2013 and then earned one or more additional 
credentials within two years. The results are displayed for three race/ethnicity groups (black, Hispanic, and 
white). Calculations were based on the following numbers of certificate-earners by race/ethnicity and field: 
2,604 (black, health care), 605 (black, MET), 186 (black, IT), 551 (Hispanic, health care), 130 (Hispanic, MET), 
36 (Hispanic, IT), 23,424 (white, health care), 5,030 (white, MET), and 1,616 (white, IT). 


your institution are less likely to return for additional postsecondary credentials 
and the degree to which racial disparities vary across students who earn their 
certificates in different fields. This will require analysis of administrative data.'4 
Administrative data can be used to explore whether student choices about which 
fields to pursue might be driving some of the disparities. You would also like 

to understand the degree to which there are racial and ethnic disparities in the 
awareness of stackable credential opportunities, plans to stack credentials, and 
the barriers that the students face in continuing on with postsecondary education. 
For these questions, you decide to collect some new data through surveys and 
focus groups. 


After collecting this evidence, your team finds that patterns mirror the histori- 

cal statewide trends, with black certificate-earners less likely to go on to earn 
additional credentials. But administrative data indicate that black students were 
more likely than white students to enroll in programs with strong opportunities for 
stacking, suggesting that program selection might not explain the racial dis- 
parities in credential stacking. Survey data suggest, however, that black certifi- 
cate-earners were less aware of their opportunities to stack credentials, and that 
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Table 2. Example of a Plan for Gathering Initial Evidence to Better Understand Racial Disparities 
in Stackable Credential Pipelines 


Data Sources Used to Address 
Questions You Would Like to Answer Questions 


How do the rates of credential stacking vary by race/ethnicity among stu- Administrative data 
dents who complete certificates in various fields? 


Are black students more or less likely to pursue fields where there are Administrative data 
opportunities for stacking credentials? 


To what degree are students aware of their opportunities to stack creden- _| Student survey 
tials, and how does this vary by race/ethnicity? 


Does the proportion of certificate-earners who plan to return to earn addi- | Student survey 
tional credentials at some point vary by race/ethnicity? 


What are the biggest barriers to returning to stack credentials? Do these Focus groups with students, 
barriers vary by race/ethnicity? faculty, and advisers 


students of all races who were less informed about stackable credential opportunities 
were less likely to report plans to return for additional credentials. Your focus groups 
with students reveal a wide variety of barriers that they face that might keep them from 
reenrolling. After completing this process, the team decides that it will definitely push 
forward with improvement related to ensuring racial equity in the stackable credential 
pipeline, and it now has a much better sense of what types of factors might be driving 
racial disparities in credential stacking. 


Step 2: Focus on the Root Causes to Identify Specific Areas 
for Improvement 

It is now time to map out the various root causes that might be driving racial dis- 
parities in credential stacking. You pull together a broad variety of individuals who 
might have insights into this issue, including faculty, advisers, and black certificate 
students. This group works collaboratively to engage in root cause analysis and 
produces a fishbone diagram that describes a wide variety of factors that might be 
contributing to these racial disparities (Figure 4). The group members then ask them- 
selves, “Which root causes are the biggest contributors to the issue, and where do 
we have leverage for change?” Drawing on the evidence collected during Step 1 and 
the perspectives of those involved in root cause analysis, your team settles on a spe- 
cific problem of practice that it would like to tackle through its improvement efforts: 
improving the information that students receive and their awareness of stackable 
credential opportunities. 
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Figure 4. Example of a Fishbone Diagram Describing the Issue of Disparities in Credential 
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Step 3: Determine What Changes Will Be Tested and 
How Progress Will Be Measured 

Now that you have identified information and awareness deficits as your problem 
of practice, the next step is to develop your model for improvement. You and your 
improvement team first work collaboratively to determine a more specific aim for 
the improvement efforts. Ultimately, you are aiming to eliminate racial differences 
in awareness of stackable credential opportunities within 18 months (Table 3). 


Next, you and your team brainstorm some potential measures of improvement 
(Table 3). Student awareness of stackable credential opportunities is an important 
measure to have on this list, given that this measure is your primary driver and 
your aim statement was structured around it. To determine whether disparities 

in student plans to stack credentials remain even when disparities in awareness 
are reduced, you also decide to measure student plans to continue to return for 
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Table 3. A Model for Improvement for Addressing Racial Disparities in Awareness of Stackable 


Credential Opportunities 


Aim: What are you trying to 
accomplish? 


We would like to eliminate racial differences in awareness of stackable credential 
opportunities within 18 months. 


Measures: How will you 
know that a change is an 
improvement? 


Student plans to reenroll to earn additional credentials 
Student awareness of follow-on opportunities 
Student satisfaction with the information and advising received 


Student use of advising sessions, the website, and other informational 
resources 


Changes: What changes will 
you make that will result in 
improvement? 


Mandatory advising sessions in the last month before completing certificate 
programs to inform students about stackable credential opportunities in 
their field 


Improvements to the website to more prominently display program and 


career maps 


additional credentials after completion of the certificate. Ideally, you would measure 
actual reenrollment and stacking, but there will not be sufficient time to track these 
longer-term measures in short improvement cycles. Finally, you want to understand im- 
provement in terms of your secondary drivers, including how often and where students 
are accessing information sources on stackable credential programs and their satisfac- 
tion with those informational resources. 


Your team identifies two initial changes that it would like to test out as possible improve- 
ments. The first is a requirement that all students meet with an adviser in the last month 
before earning their initial certificate to learn about stackable credential opportunities in 
their field and potentially consider reenrolling to continue with additional credentials. The 
concern is that, historically, advising has relied on visits that are initiated by students, and 
black students have been less likely to visit the advising office and less likely to inquire 
about opportunities to stack credentials. The improvement team searches the literature 
and finds that mandatory “intrusive” advising at key periods within a program can help 
build student awareness of key information and improve student success outcomes.!° The 
team chooses improvements to the website information on stackable credential programs 
as the second change that it would like to test. In the first PDSA cycle, the improvement 
team decides to focus on the intrusive advising change. 


Step 4: Test, Monitor, and Refine Improvements Through 
an Iterative Process 

Now that you have narrowed in on a specific change that will be tested to improve 
racial disparities in awareness of stackable credential opportunities, you are ready to 
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carry out a PDSA cycle. In the PDSA cycle, your team plans to identify a group 

of students who are close to completing their certificate across two different 
programs and require those students to participate in mandatory advising in their 
last month before completing a certificate. You will then examine data on these 
students to understand more about whether these mandatory advising sessions 
lead to improvements in awareness of stackable credential opportunities, as well 
as whether the changes should be scaled across the institution. The following 
subsections briefly describe how this cycle might be carried out. 


Plan 

First, you set out to plan for your PDSA cycle. A major part of this planning effort 
is to identify questions, make predictions, and determine which data will be 
used to examine those predictions. The measures you identified in Step 3 serve 
as a starting point for developing these questions and predictions. Ultimately, 
you identify seven predictions that will be explored through the PDSA cycle 
(Table 4). If the change to advising is successful, you predict that large percent- 
ages of students assigned to mandatory advising sessions will attend them, and 
you anticipate that most of those sessions will provide information on stackable 
credential opportunities. Your team identifies advising records as the best data 
source for assessing these predictions. A second set of predictions focus on 
comparisons between black students and white students, anticipating that there 
will be no racial disparities in whether students participate in mandatory advis- 
ing sessions, whether those sessions cover stackable credential opportunities, 
whether students are satisfied with the advising they receive, and whether gaps 
in awareness and interest will decrease. Finally, you predict that students who 
are assigned to mandatory advising will be more likely to demonstrate aware- 
ness of follow-on programs and more likely to state plans to pursue additional 
credentials relative to students who are not assigned to mandatory advising. 


Next, you and your team sketch out all of the activities that will take place in the 
PDSA cycle and assign roles and responsibilities and timelines for those activities 
(Table 5). The activities include those required to roll out a new mandatory advis- 
ing requirement, as well as those required to collect data and assess whether the 
new advising requirement was an improvement. 


Do 

Next, you roll out your new mandatory advising requirement with the selected 
group of students who are in certificate programs that offer opportunities for 
stacking credentials. You then collect your survey and advising data according 
to the plan set out in Table 5. A new two-page guide with requirements and tips 
for the advising session is created by the advising lead on your improvement 
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Table 4. Example of Questions, Predictions, and Data Sources That Could Be Used to Assess a 
New Mandatory Advising Requirement 


Question 


Prediction 


Data Source 


Are black students more or less 
likely to meet with advisers when 
mandatory advising sessions are 
established? 


90 percent of students who are selected to receive 
mandatory advising will see an adviser. 


Advising records 


Are stackable credential oppor- 
tunities discussed at mandatory 
advising sessions? 


At least 95 percent of mandatory advising sessions 
will discuss topics related to stackable credential 
opportunities. 


Advising records 


Do findings on participation in 
advising sessions and topics of dis- 
cussion differ by the race/ethnicity 
of the student? 


There will be no differences in the likelihood of 
meeting with an adviser and discussing stackable 
credential opportunities between black and white 
students who are selected to receive mandatory 
advising. 


Advising records 


Are there differences in satisfaction | There will be no differences in satisfaction with Survey 
with advising sessions by the race/ | advising between black and white students who are 

ethnicity of the students? selected to receive mandatory advising. 

Are students who are asked to Students who are selected to receive mandatory Survey 
participate in advising sessions advising will be more likely to identify one follow-on 

more likely to be aware of stackable | program than students who are not targeted with 

credential opportunities? mandatory advising. 

Are students who are asked to par- | Students who are selected to receive mandatory Survey 
ticipate in advising sessions more advising will be more likely to state plans to earn 

likely to plan to pursue stackable additional credentials than students who are not 

credential opportunities? targeted with mandatory advising. 

Are gaps between black and white | Gaps between black and white students will be Survey 


students in awareness of stackable 
credential opportunities and plans 
to pursue them smaller among 
students who participate in advising 
sessions? 


smaller among those who attended advising ses- 
sions. 


team using feedback from advisers. All advisers who will deliver mandatory advis- 


ing sessions are provided with this guide and required to participate in a one-hour 


training that describes opportunities to stack multiple credentials in the selected 
programs, career advising, and other topics that might be covered in these sessions. 


Advising sessions are held over a five-week period in the middle of the semester. 


Advisers keep records of every advising session in a case management data system, 


documenting who visited and what was discussed. You prepare a short five-question 
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Table 5. Example of Activities, Roles and Responsibilities, and Timelines for a PDSA Cycle 
Focused on Testing a New Mandatory Advising Requirement 


Activity Lead (other participants) Deadline 
Prepare guidance and training for Advising administrator (program lead) Day 5 
advisers on new sessions 

Select students who will be assigned | Improvement team lead (institutional re- Day 8 
mandatory advising sessions search staff, program leads) 

Train advisers Advising administrator (lead advising staff, Day 10 


advisers for two pilot programs) 


Ensure that the appropriate advising | Advising administrator (advisers for two pilot | Days 10-45 


data are being collected programs) 

Inform students about mandatory Advising administrator (advisers for two pilot | Day 12 
advising sessions programs) 

Implement mandatory sessions and Advising administrator (advisers for two pilot | Days 13-45 
assist with troubleshooting programs) 

Prepare survey questions Improvement team lead (institutional re- Day 20 


search staff, program leads) 


Administer survey Faculty lead (faculty) Day 50 


Assess results and document findings | Improvement team lead (improvement team | Day 68 
members, institutional research staff) 


Share results and gather input Improvement team lead (advisers, faculty, Day 80 
students) 

Decide whether and how to scale, Improvement team lead (improvement team | Day 85 

refine, or abandon the change members, other key stakeholders) 


Identify new issues and questions for | Improvement team lead (improvement team | Day 85 
improvement efforts members) 


student survey, which is administered by faculty who teach common final-se- 
mester courses for the two programs after advising sessions wrap up. 


Study 

It is now time to assess the results, share them to gather input from a wide variety 
of stakeholders, and record the findings. You find that most of the students who 
are asked to attend mandatory advising sessions do end up attending these ses- 
sions and that participation levels and advising experiences are similar for black 
and white students. Students who are assigned to mandatory advising sessions 
demonstrate greater levels of awareness of follow-on programs and plans to 

earn additional credentials. These findings align with your predictions, providing 
evidence that the change was an improvement. However, you find that at least 
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25 percent of advising sessions did not touch on stackable credential programs; this 
number is below the target you set. This suggests that refinements to guidance and 
training might be necessary to ensure a higher level of compliance with the expected 
advising session requirements. You share these findings broadly with advisers, faculty, 
and students to hear their interpretations of the findings, solicit input to inform decision- 
making on next steps, and familiarize them with evidence supporting the mandatory 
advising requirements to build buy-in. 


Act 

Now that you have assessed the evidence, it is time to make some decisions. First, will 
you scale the improvement to all students and additional programs or abandon the 
change and test out a new one? The evidence seems to point to mandatory advising 
sessions being an improvement, so you and the team decide to scale these sessions 

to a larger number of students. A second question is whether to refine the mandatory 
advising approach before scaling. Findings from the analysis suggest that too many 
sessions lacked discussion of stackable credential opportunities, so you decide to refine 
the guidance materials and training to place greater emphasis and urgency on discuss- 
ing stackable credential opportunities with students and making connections between 
these efforts and their aim to support student success and equity. The team will contin- 
ue to assess these efforts to refine and scale mandatory advising sessions through a 
second PDSA cycle, repeating Step 4 until the change is successfully scaled. Finally, you 
and your team document other issues and questions related to the stackable credential 
pipeline that might be addressed through future PDSA cycles. 


Your team continues to engage in PDSA cycles, testing and scaling improvements to the 
information students receive on stackable credential opportunities. After 18 months, 
you achieve your aim of eliminating racial disparities in awareness of stackable creden- 
tials, and your team moves onto other areas where the stackable credential pipeline 
could be improved. 


Challenge 2: Lower Rates of Stacking Among Students 
Who First Earn Certificates at Ohio Technical Centers 


A recent RAND report suggests that students who earned certificates at OTCs were 

less likely to return and complete additional postsecondary credentials than were 
students who earned certificates at other types of institutions.'° This finding serves as 
the starting point for a fictional example in which a group of postsecondary institutions 
carries out the four-stage improvement process to make improvements to the stackable 
credential pipeline and address lower rates of stacking among OTC certificate-earners. 
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Figure 5. Percentage of First-Time Certificate-Earners Who Stacked One or More Additional 
Credentials Within Four Years, by Field and Type of Institution at Which the Initial Certificate 
Was Earned 
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SOURCES: This figure was drawn from Daugherty et al., 2020. Data for the figure were drawn from HEI and OTC 
data files in the Ohio Longitudinal Data Archive (Ohio Longitudinal Data Archive, undated). 

NOTE: The figure presents the percentage of individuals who earned an initial certificate in each of the three fields 
and overall (three fields combined) between 2005 and 2011 and then earned one or more additional credentials 
within four years. The results are displayed for three types of institutions (OTC, community college, and university). 
Calculations were based on the following numbers of certificate-earners by institution type and field: 4,086 (OTC, 
health care), 274 (OTC, MET), and 71 (OTC, IT), 16,179 (community college, health care), 4,440 (community college, 
MET), and 1,318 (community college, IT), 745 (university, health care), 162 (university, MET), and 234 (university, IT). 


Step 1: Assess Evidence in Your Own Context to Identi- 
fy Issues in the Stackable Credential Pipeline to Tackle 
Using statewide data, RAND researchers found that 16 percent of the students 
who earned an initial certificate in health care, MET, or IT at an OTC between 
2005 and 2013 went on to earn an additional credential within two years, com- 
pared with 39 percent of certificate-earners at community colleges (Figure 5).'” 
The disparities in the rates of stacking between those starting at OTCs and those 
starting at community colleges were particularly large for MET and IT fields. This 
suggests that there might be issues in the stackable credential pipeline that are 
acting as barriers for students who complete certificates at OTCs in particular. 


You are leading a task force in your region that is made up of representatives from 
the local OTC, community college, and university in your region, with the goal of 
working collaboratively to build stronger stackable credential pipelines across the 
institutions. After looking at the statewide findings on the lower rates of stacking 
among OTC certificate-earners, you would like to consider this as a potential 

area for improvement in your region. Before you dive into improvement efforts to 
address this issue, the first step is to gather some additional evidence to better 
understand the issue within your own context. 
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Table 6. Example of a Plan for Gathering Initial Evidence to Better Understand Issues in the 


Stackable Credential Pipeline for OTC Certificate-Earners 


Questions You Would Like to Answer 


Data Sources Used to Address 
Questions 


Are students who complete certificates at OTCs in our region less likely to 
return for additional credentials than students who complete certificates at 
the region’s community colleges? 


Administrative data 


To what degree do differences in student characteristics (e.g., age, 
economic disadvantage) and field choice across institutions account for 
differences in rates of stacking? 


Administrative data 


Is there variation in the degree to which students earning certificates at 
different types of institutions are aware of the follow-on programs available 
to them? 


Student survey 


How do plans to reenroll in additional credentials, work, or do both vary 
across certificate-earners at different types of institutions? 


Student survey 


Are students aware of articulation agreements, such as Career-Technical 
Assurance Guides, the One-Year Option, and bilateral agreements for cer- 
tain programs? Have they made use of these options? 


Student survey, administrative 
data 


What are the biggest barriers to returning to stack credentials for OTC 
certificate-earners? Do these barriers differ from those who complete 
certificates at a community college? 


Student survey 


Are there strong regional articulation agreements in place? What other 
ways do institutions support transitions across institutions? How do 
institutions think about one another’s roles in the pipeline and the 
opportunities for improving pipelines? 


Articulation agreements, inter- 
views with administrators 


Your team members work collaboratively to identify seven questions that they would 


like to answer to better understand the issues related to the stackable credential 
pipeline for OTC certificate-earners, as well as data sources that can be used to address 


these questions (Table 6). You first want to understand whether your region sees simi- 


lar disparities in stacking for OTC certificate-earners, and you propose to use adminis- 
trative data to examine this question. Your other questions explore some of the possible 


reasons for these disparities, such as sorting of different types of students into different 


institutions, differences in awareness of stackable program offerings, and barriers to 
transferring credit. Administrative data can be used to look at whether differences in 


student characteristics explain variation in rates of stacking and whether students are 


utilizing articulation opportunities. You and your team decide to conduct a brief survey 
with students in similar programs at the OTC and community college in your region to 


learn about differences in awareness of stackable program offerings; awareness of ar- 
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ticulation opportunities; plans to reenroll, seek employment, or do both post—cer- 
tificate completion; and the other types of barriers students face when reenrolling 
to earn additional credentials. Finally, a review of cross-institution agreements 
and programs and interviews with administrators can shed light on the degree 

to which infrastructure and perspectives of key stakeholders are aligned with 
supporting cross-institution transfer. 


After collecting this evidence, your team finds that patterns mirror the historical 
statewide trends, with OTC certificate-earners in the region less likely to go on 

to earn additional credentials. Your analysis of student characteristics suggests 
that the sorting of different types of students into different types of institutions 
explains some of the differences, but you still find disparities in rates of credential 
stacking among similar students in the same programs at the OTC and commu- 
nity college. Your survey data identify several potential barriers: OTC certificate 
enrollees are less likely than community college students to be aware of stack- 
able credential opportunities, and they know relatively little about the articulation 
agreements in place to help them transfer credits. OTC students are also more 
likely to report plans to work before pursuing additional credentials, and students 
who directly enter the workforce after earning a certificate report that it can 

be challenging to reconnect with educational institutions after being out in the 
workforce for a few years. And although the institutions have articulation agree- 
ments in place, little is being done by any institution to encourage students to 
take advantage of these agreements. After completing this process, you and your 
team decide that you will definitely push forward with improvement related to 
addressing issues in the stackable credential pipelines for OTC certificate-earners, 
and you have now developed a better sense of some of the barriers that these 
students face. 


Step 2: Focus on the Root Causes to Identify Specific 
Areas for Improvement 

It is now time to map out the various root causes that might be preventing OTC 
certificate-earners from returning to earn additional credentials. You pull together 
a broad variety of individuals who might have insights into this issue, including 
faculty, advisers, and students from various institutions in the region, as well as 
representatives from major employers in the region. This group works collabo- 
ratively to engage in root cause analysis and produces a fishbone diagram that 
describes a wide variety of factors that might be contributing to lower rates of 
stacking among OTC certificate-earners (Figure 6). The group members then 
ask themselves, “Which root causes are the biggest contributors to the issue, 
and where do we have leverage for change?” Drawing on the evidence collected 
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Figure 6. Example of a Fishbone Diagram Describing the Issue of Lower Rates of Credential 
Stacking Among OTC Certificate-Earners 
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during Step 1 and the perspectives of those involved in root cause analysis, your team 
decides to focus its efforts on improving the information that students receive and their 
awareness of stackable credential opportunities. 


Step 3: Determine What Changes Will Be Tested and How 
Progress Will Be Measured 

Now that you have identified information and awareness barriers among OTC certifi- 
cate-earners as your problem of practice, the next step is to develop your model for 
improvement. You and your improvement team first work collaboratively to determine 
a more specific aim for the improvement efforts. Ultimately, you decide that you are 
aiming to ensure that the majority of students who have completed certificates at the 
OTC are aware of follow-on credential options and articulation agreements, and you 
set a timeline of two years to achieve this aim (Table 7). 
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Table 7. A Model for Improvement for Increasing Awareness of Stackable Credential Programs 
and Articulation Opportunities Among OTC Certificate-Earners 


Aim: What are you trying to 
accomplish? 


We would like 90 percent of certificate-earners at the OTC to be aware of their 
follow-on credential options and the articulation agreements that will help them 
get credit for their prior coursework within two years. 


Measures: How will you 
know that a change is an 
improvement? 


e Student interest in continuing to stack credentials 

e Student awareness of follow-on opportunities 

e Student awareness of articulation agreements 

e Student satisfaction with the information and advising received 

e Student engagement with advisers and other informational resources 


Changes: What changes will 
you make that will result in 
improvement? 


e Career and transfer handbooks that are provided to OTC students describ- 
ing all of the follow-on credential opportunities available in the region and 
providing tips on ensuring the articulation of credit 


e Periodic on-site “drop-ins” from community college and university represen- 
tatives to OTC classes to provide information on their programs and articula- 
tion agreements and assist students with applications 


e Informational workshops hosted by major employers in the region to pro- 
vide information to OTC certificate-earners who have been in the workforce 
for a while 


Next, you and your team brainstorm some potential measures of improvement 
(Table 7). Student awareness of follow-on credential opportunities and articu- 
lation agreements are your primary drivers and important measures to have on 
this list, given that your aim statement was structured around these measures. To 
determine whether you have moved the needle on student plans to stack cre- 
dentials when disparities in awareness are reduced, you also decide to measure 
student interest in reenrolling after completion of the certificate. Finally, you want 
to examine your secondary drivers, looking at how often and where students are 
accessing information sources on stackable credential programs and their satis- 
faction with those informational resources. 


You and your team look to the literature for promising practices aimed at building 
awareness around stackable credential opportunities. You also gather ideas from 
the broad group of stakeholders involved in the root cause analysis. You ultimate- 
ly identify three changes that you would like to test out as possible improvements. 
The first is requiring that all OTC students receive a career and transfer handbook 
when they enter a certificate program. This should ensure that all students receive 
detailed information on their options. The second change that you would like to 
test is inviting advisers from the community college and university in the region 

to conduct periodic informational sessions on their programs and their credit 
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articulation opportunities. Your hope is that exposure to other institutions in the region 
and the opportunity to build excitement about transferring into programs at other insti- 
tutions will increase the likelihood that students decide to pursue additional credentials. 
The third change that you would like to test is providing workshops at employers in the 
region who employ a large number of former OTC certificate-earners. These workshops 
aim to provide information on follow-on and articulation opportunities to individuals 
who have been in the workforce for a while since completing their OTC certificate. 
These sessions can also help inform employers about the benefits of encouraging 

their employees to stack credentials. You and the task force decide to develop anoth- 
er improvement team that will conduct PDSA cycles for the handbook and the on-site 
informational sessions, while your team pushes forward with testing out the employer 
workshops as a Strategy for improvement. 


Step 4: Test, Monitor, and Refine Improvements Through 
an Iterative Process 

Now that you have narrowed in on a specific change that will be tested to improve 
awareness of stackable credential programs and articulation opportunities among OTC 
students, you are ready to carry out a PDSA cycle. In the PDSA cycle, your team plans to 
identify two local companies that employ a substantial number of OTC graduates (e.g., 
a hospital system and a manufacturing plant) and conduct workshops at each of these 
organizations. You will then examine data on the individuals targeted at these organi- 
zations to understand more about whether these workshops lead to improvements in 
awareness of stackable credential opportunities, as well as whether the changes should 
be scaled to other employers. The following subsections briefly describe how this cycle 
might be carried out. 


Plan 

First, you set out to plan for your PDSA cycle. A major part of this planning effort is to 
identify questions, make predictions, and determine which data will be used to exam- 
ine those predictions. The measures you identified in Step 3 serve as a starting point 

for developing these questions and predictions. Ultimately, you identify six predictions 
that will be explored through the PDSA cycle (Table 8). If the informational workshops 
are successful, you predict that at least 100 employees and ten supervisors or human 
resources representatives will attend the sessions, and your team collects attendance at 
sessions to measure this. Pre- and post-workshop surveys are used to measure changes 
in awareness of stackable credential opportunities and interest in pursuing these oppor- 
tunities among those who participate, and post-workshop surveys also assess participant 
satisfaction. Finally, you decide to conduct interviews with employer leadership after the 
sessions to determine whether the workshops led employers to introduce other initia- 
tives or Supports to encourage their employees to seek out additional credentials. 
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Table 8. Example of Questions, Predictions, and Data Sources That Could Be Used to Assess a 
New Employer-Based Workshop 


Question Prediction Data Source 
How many (and which types of) indi- | The workshops will reach at least 100 employees Attendance 
viduals participate in the workshops? | and at least ten supervisors or human resources sheets 


representatives across the two organizations. 


Did participants in the workshop per- 


At least 75 percent of participants will report that 


Post-workshop 


ceive the information as being helpful | the information was helpful and felt relevant. participant 

and relevant to them? survey 

Do the workshops help ensure that The percentage of participants who are able to Pre- and 

individuals are aware of stackable demonstrate knowledge of follow-on programs and | post-workshop 

credential opportunities? articulation agreements is higher after the work- participant 
shop. surveys 

Do the workshops increase the likeli- | The percentage of employee participants who Pre- and 

hood that students will plan to pursue | express interest in enrolling in stackable credential | post-workshop 

stackable credential opportunities? programs is higher after the workshop. participant 

surveys 
Do the workshops enhance the efforts | Representatives from both employers will describe | Interviews 


of employers to encourage their em- 
ployees to seek out credentials? 


at least one new effort to support and encourage 
their employees to pursue additional credentials. 


Next, you and your team sketch out all of the activities that will take place in 


the PDSA cycle and assign roles and responsibilities and timelines for those 

activities (Table 9). The activities include those required to develop and provide 
the workshops, as well as those required to collect data and assess whether the 
workshops were an improvement. 


Do 


Next, you roll out your workshop sessions at both of the selected employers and 
collect your survey and advising data according to the plan set out in Table 8. 


You select health and manufacturing program administrators to develop presen- 


tation slides, activities, worksheets, and informational handouts for a one-hour 


workshop session that is tailored to each of the different employers. You then 
host these workshops over a two-week period. You collect attendance sheets 
and pre- and post-workshop surveys at the time of these workshops and follow 
up with employers approximately three weeks after the workshops are held to 
conduct interviews. 
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Table 9. Example of Activities, Roles and Responsibilities, and Timelines for a PDSA Cycle 


Focused on Testing a New Employer-Based Workshop 


Activity Lead (other participants) Deadline 

Identify the employer sites where the | Improvement team lead (improvement team | Day 8 

workshops will be provided members, career services rep) 

Prepare a plan and supporting materi- | Health and manufacturing program admin- Day 19 

als for the workshop istrators (faculty and advising leads from 
various institutions) 

Inform employers about who should Health and manufacturing program adminis- | Day 22 

attend and what will take place trators 

Prepare survey and interview ques- Institutional research representative Day 25 

tions (improvement team members, program 
administrators, advising leads) 

Host workshops, collect attendance, | Health and manufacturing program adminis- | Days 28-40 

administer pre- and post-workshop trators and advising leads 

surveys 

Conduct interviews Health and manufacturing program adminis- | Day 70 
trators 

Assess results and document findings | Improvement team lead (improvement team | Day 82 
members, institutional research staff) 

Share results and gather input Improvement team lead (advisers, faculty, Day 90 
students, employers) 

Decide whether and how to scale, Improvement team lead (improvement team | Day 96 

refine, or abandon the change members, other key stakeholders) 

Identify new issues and questions for | Improvement team lead (improvement team | Day 96 


improvement efforts 


members) 


Study 


It is now time to assess the results, share them to gather input from a wide variety of 


stakeholders, and record the findings. You find that you surpassed your attendance 


requirements, with more than 200 employees participating in the workshops. Looking 


at the survey data, you find that there are two groups of employees: Half of the employ- 


ees found the information very useful and saw big gains in awareness and interest in 


pursuing credentials, while the other half was less interested in the information and did 


not seem to pick up much from the sessions. This suggests that the workshops were 


an improvement, but you might want to consider providing more guidance to employ- 


ers and employees to ensure that the workshops are targeted to employees who are 
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most likely to be interested in them. You also find that, despite your predictions, 
employers did not make any additional efforts to encourage and support employ- 
ees interested in pursuing stackable credentials. You share these findings broadly 
with advisers, faculty, students, and employers to hear their interpretations of the 
findings, solicit input to inform decisionmaking on next steps, and familiarize them 
with evidence to build buy-in. 


Act 

Now that you have assessed the evidence, it is time to make some decisions. 
First, will you scale the workshop to additional employers or abandon the change 
and focus on other changes that you want to test? Some of the evidence points to 
workshops being an improvement, so you and the team decide to hold these ses- 
sions annually and scale them to new employers. A second question is whether to 
refine the workshops before scaling. Findings suggest that the workshops might be 
useful for only a subset of employees, so you and your team develop guidance for 
employers on identifying those employees and targeting the sessions to them. The 
team will continue to assess these expanded and refined workshop efforts through 
a second PDSA cycle, repeating Step 4 until the change is successfully scaled. 


Although these workshops were not successful in getting employers to develop 
new initiatives to support their employees in returning to pursue educational cre- 
dentials, you decide that this remains an important barrier to bringing in returning 
students. So, you decide to tackle this issue separately and end up developing a 
different type of informational session and materials that are explicitly targeted to 
supervisors and human resources staff. 


After two years, you achieve your aim of increasing awareness of stackable cre- 
dentials opportunities among certificate-earners at OTCs, and your team moves 
onto other areas where the stackable credential pipeline could be improved. 


Challenge 3: Excess Credit Hours Among Students 
Who Stack Credentials 


A recent RAND report suggests that students who stack credentials might be 
earning excess credit hours.'* This finding serves as the starting point for a 
fictional example in which a postsecondary institution carries out the four-stage 
improvement process to make improvements to its stackable credential pipeline 
and address issues related to excess credit hours. 


Step 1: Assess Evidence in Your Own Context to Identi- 
fy Issues in the Stackable Credential Pipeline to Tackle 
Using statewide data, RAND researchers found that students who completed a 
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Table 10. Cumulative Number of Credit Hours Earned for Associate’s Degree—Earners, by Field 
and Whether the Degree Was Stacked with a Certificate 


Credit Hours Earned 


Certificate and 


Field Associate’s Degree Associate’s Degree Only Difference 
Overall 102.9 85.8 171 
Health care 108.6 87.7 20.9 
MET 9525) 81.4 12.1 
IT 93.7 80.4 13.3 


SOURCES: This table was adapted from Daugherty et al., 2020. Data for the table were drawn from HEI and OTC data files 
in the Ohio Longitudinal Data Archive (Ohio Longitudinal Data Archive, undated). 


NOTE: The values in this table represent the average number of credit hours earned and the average number of terms of 
enrollment for (1) individuals who earned an initial certificate in one of the three fields between 2005 and 2011 and then 
earned an associate’s degree within two years and (2) individuals who earned only an associate’s degree in one of the 
three fields (i.e., did not stack on top of a certificate) in the same time frame. Calculations for individuals who earned a 
certificate and an associate’s degree are based on the following numbers of students: 5,278 (health care), 2,318 (MET), 
and 999 (IT). Calculations for individuals who earned an associate’s degree only are based on the following numbers of 
students: 62,958 (health care), 17,541 (MET), and 8,062 (IT). 


certificate and associate’s degree earned an average of 17 additional credits when com- 
pared with students who earned only an associate’s degree (Table 10).'° This suggests 
that certificate and associate’s degree programs that were designed to be stackable 
might not articulate as well as institutions would hope. In addition to costing students 
time and money as they pursued additional credit hours, these inefficiencies might have 
prevented students from attempting or completing multiple credentials. 


You are leading a task force at your institution charged with improving stackable cre- 
dential pipelines in common technical fields. After looking at the statewide findings on 
excess credit hours, you would like to consider this as a potential area for improvement 
in your institution. Before you dive into improvement efforts to address this issue, the 
first step is to gather some additional evidence to better understand the issue within 
your own context. 


Your team members work collaboratively to identify six questions that they would like 
to answer to better understand the issues in the stackable credential pipeline relat- 

ed to excess credit hours, as well as data sources that can be used to address these 
questions (Table 11). First, your team has questions about whether students stacking 
credentials in your institution are earning excess credit hours (and in which programs), 
and you plan to use administrative data to answer this question. You are also interested 
in what types of factors might be contributing to students earning excess credit hours. 
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Table 11. Example of a Plan for Gathering Initial Evidence to Better Understand Excess Credit 


Hours Among Students Who Stack Credentials 


Questions You Would Like to Answer 


Data Sources Used to Address 
Questions 


What is the average number of credit hours earned among associate’s 
degree students who did and did not complete certificates across various 
fields? 


Administrative data 


What proportion of credit hours were earned outside the core program 
areas? 


Administrative data 


What are the total numbers of credit hours required for completion of 
common certificate and associate’s degree stacks? 


Documentation on program 
course requirements 


Do students face challenges in getting approval for credits from their 
certificate program to articulate into their associate’s degree? 


Student and adviser focus groups 


Do students face challenges with getting into courses they need to wrap up 
a degree? 


Student and adviser focus groups 


What other barriers might lead to excess credentials? Do the barriers that 
students face vary across fields? 


Student, adviser, and faculty 
focus groups 


You plan to examine program documentation to understand how many credits are 
required for completion of various stackable credentials and determine whether 
these programs have been structured in ways that require excess credit hours. 
You also decide to conduct student and faculty focus groups and ask about 
various barriers that students face, including issues with articulating credits and 
getting into the necessary courses for graduation. 


After collecting this evidence, your team finds that patterns mirror the historical 
statewide trends, with approximately 20 additional credit hours accumulated 

by students who earn a certificate prior to the associate’s degree. Your review 
of program course requirements reveals wide variation in the degree to which 
programs are built to be stackable; some programs allow all of the certificate 
credit hours to be used toward the degree, while other programs allow less than 
50 percent of certificate credit hours to count toward the degree. You find that 
health programs in your institution tend to be designed to be more stackable, 
while MET and IT certificate and associate’s degree programs tend to share 
fewer credit requirements. In the focus groups, students and advisers did not 
report any issues with getting approvals to articulate credits. However, students 
and advisers reported that oversubscribed courses and programs were problem- 
atic, especially in health care fields. In addition, the nursing program required 
completion of a nurse’s aide certificate as a prerequisite for some students who 
could not qualify based on grades, but the program did not award credit for this 
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Figure 7. Example of a Fishbone Diagram Describing the Issue of Excess Credit Hours 
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prior coursework. After completing this process, the team decides that it will definitely 
push forward with improvement related to addressing inefficiencies in the stackable 
credential pipeline, and the team now has a much better sense of what types of fac- 
tors might be driving these issues. 


Step 2: Focus on the Root Causes to Identify Specific Areas 
for Improvement 


It is now time to map out the various root causes that might be driving excess credit 
hours. You pull together a broad variety of individuals who might have insights into this 
issue, including faculty, advisers, and certificate students. This group works collabora- 
tively to engage in root cause analysis and produces a fishbone diagram that describes 
a wide variety of factors that might be contributing to excess credit hours (Figure 7). 
The group members then ask themselves, “Which root causes are the biggest con- 
tributors to the issue, and where do we have leverage for change?” Drawing on the 
evidence collected during Step 1 and the perspectives of those involved in root cause 
analysis, your team decides to pursue different improvement strategies for different fields. 
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Table 12. A Model for Improvement for Addressing Excess Credit Hours Among Students Who 


Stack Credentials 


Aim: What are you trying to 
accomplish? 


We would like to ensure that all certificate programs deemed “stackable” allow 
for at least 80 percent of all certificate credit hours to count toward the degree 
within two years. 


Measures: How will you 
know that a change is an 
improvement? 


e The number of credits earned that do not articulate into a degree program 
e The degree to which course requirements are overlapping 

e Faculty satisfaction with program requirements 

e Adviser and student awareness of course requirements 


Changes: What changes will 
you make that will result in 
improvement? 


e Audit of and revisions to program requirements 


e Pushing out of information on new program requirements to students and 
advisers 


In health care, excess credit hours seem to be driven in large part by oversubscribed 
programs (i.e., too many students competing for too few slots) and high demand for 
courses that prevent students from enrolling in required courses for the major. In 
the meantime, these students end up enrolling in unnecessary courses to fill their 
course schedules and retain their financial aid. So, the improvement team pursues 
improvement efforts to address this issue in health fields. In manufacturing and IT, 
evidence suggests that limited overlap in course requirements for certificate and 
associate’s degree programs might be contributing to excess credit hours. This 
example focuses on the manufacturing and IT improvement efforts. 


Step 3: Determine What Changes Will Be Tested and 
How Progress Will Be Measured 

Now that you have selected limited overlap in program course requirements as 
your problem of practice, the next step is to develop your model for improvement. 
You and your improvement team first work collaboratively to determine a more 
specific aim for the improvement efforts. Ultimately, you all decide that you are 
aiming to ensure that within two years, at least 80 percent of all required credit 
hours for each of the stackable certificate programs will be able to articulate into 
an associate’s degree (Table 12). 


Next, you and your team brainstorm some potential measures of improvement 
(Table 12). The degree to which coursework overlaps across certificate and asso- 
ciate’s degree programs is an important measure to have on this list, given that 
this measure is your primary driver and your aim statement was structured around 
it. Although you would like to look at actual credit accumulation before and after 
the change, there might not be time to measure these changes within shorter 
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improvement cycles. In addition, your team is concerned that efforts to streamline 
course requirements will lead to the elimination of important coursework and program 
requirements, as well as to frustration among faculty, so you decide to measure faculty 
satisfaction with program requirements. Finally, it is important that these new course 
requirements are communicated effectively to advisers and students, so you decide to 
measure adviser and student awareness of course requirements. 


After searching the literature and talking with different stakeholders, your team mem- 
bers agree that there is one change that they would like to test as a possible improve- 
ment to address this issue of limited alignment in course requirements. You decide 
that you will ask program administrators and faculty to conduct an audit of their course 
requirements for each program and make efforts to increase the degree to which 
credit hours from the certificate program count toward follow-on associate’s degree 
programs. In addition to modifying course requirements for certificate and associate’s 
degree programs, administrators and faculty must communicate any changes to ad- 
visers and students. Once you have settled on this change, your team begins a PDSA 
cycle to test whether this change was an improvement. 


Step 4: Test, Monitor, and Refine Improvements Through 
an Iterative Process 

Now that you have narrowed in on a specific change that will be tested to streamline 
and ensure overlap in program requirements, you are ready to carry out a PDSA cycle. 
In the PDSA cycle, your team plans to start with a subset of the certificate and degree 
offerings in manufacturing and IT to test out the process of auditing and streamlining pro- 
gram requirements. You will then examine data on these programs to understand more 
about whether these revisions lead to improvements in the opportunities for students 

to move more efficiently through stackable credential programs, as well as whether the 
changes should be scaled across the institution. The following subsections briefly de- 
scribe how this cycle might be carried out. 


Plan 

First, you set out to plan for your PDSA cycle. A major part of this planning effort is to 
identify questions, make predictions, and determine which data will be used to examine 
those predictions. The measures that you identified in Step 3 serve as a starting point 
for developing these questions and predictions. Ultimately, you identify five predictions 
that will be explored through the PDSA cycle (Table 13). If the changes to streamline 
program requirements are successful, you predict that the two fields piloting the 
change will conduct an audit for at least three of their certificate programs, and you 
ask programs to submit documentation on this audit process to assess this prediction. 
You also predict that course requirements will be streamlined for at least 75 percent 

of the programs audited, and you collect and compare program requirement docu- 
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Table 13. Example of Questions, Predictions, and Data Sources That Could Be Used to Assess 
Efforts to Streamline and Align Program Requirements 


Question Prediction Data Source 
Are programs audited to determine In each of the two fields, program staff will audit Audit 

the level of overlap in course course requirements for at least three certificate documentation 
requirements? programs and the follow-on degrees they stack into. 

Will programs refine course Across the programs audited and refined, atleast | Course 


requirements to increase overlapping 
course requirements? 


75 percent will have more program overlap after 
the process. 


requirement 
documentation 


Are faculty satisfied that the programs 
meet key learning objectives despite 
streamlined coursework? 


Fewer than 10 percent of faculty report that the 
key learning objectives are unlikely to be met with 
streamlined course requirements. 


Faculty pulse 
survey 


Are advisers aware of the new course 


95 percent of advisers are familiar with all of the 


End-of-training 


requirements? changes to course requirements. assessment 
Are students aware of the new 75 percent of students in focus groups will be famil- | Student focus 
learning requirements? iar with the changes to their program. groups 


mentation before and after the process to determine whether this is the case. 


To assess faculty satisfaction with the new course requirements, you send out 


a simple one-question survey, with the hope that less than 10 percent of faculty 
raises concerns about learning objectives met through the streamlined course 


requirements. You also want to make sure that substantial numbers of advisers 


and students are aware of these changes. You plan to assess adviser knowledge 
at the end of a training session, and you plan to learn about student awareness 


through focus groups. 


Next, you and your team sketch out all of the activities that will take place in the 
PDSA cycle and assign roles and responsibilities and timelines for those activities 


(Table 14). The activities include those required to roll out the new course require- 


ments, as well as those required to collect data and assess whether the streamlined 


courses were an improvement. Because it can take time to make changes to pro- 


gram requirements, the timeline is longer than the usually recommended 90 days. 


Next, you begin the process of auditing and streamlining program requirements with 


selected programs and collecting the various sources of data according to the plan 


set out in Table 14. Groups of faculty, administrators, and advisers settle on chang- 


es to program requirements, and then advisers and students are informed about 
the changes to program requirements through trainings, a new website, emails to 


students who are newly enrolling in the programs, and information in advising ses- 
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Table 14. Example of Activities, Roles and Responsibilities, and Timelines for a PDSA Cycle 
Focused on Efforts to Streamline and Align Program Requirements 


Activity Lead (other participants) Deadline 

Select fields or programs that refine- | Improvement team lead (program leads) Day 5 

ment will take place in 

Identify existing overlap across cre- Program administrator (program staff) Day 15 

dential programs 

Collect documentation on course Program leads Day 50 

requirements and audit process 

Add question to regular faculty survey | Institutional research representative (faculty | Day 50 
leads, program leads) 

Assess course documentation and Program leads (faculty, institutional research | Day 60 

faculty survey staff) 

Hold meetings with faculty, advisers, | Program administrator (faculty, advisers, Day 65 

and administrators to determine any _| other administrators) 

changes to program course require- 

ments 

Implement changes to program Program administrator (faculty, advisers, Days 66-115 

course requirements other administrators) 

Train advisers on new course require- | Advising lead (advisers) Day 95 

ments for programs 

Develop and administer adviser train- | Advising lead (program leads) Day 95 

ing assessment 

Update websites Web programmer (program administrator, Day 105 


lead adviser, lead faculty) 


Send out emails to students and share 
information in advising session 


Advising lead (advisers, students) 


Days 115-140 


Conduct focus groups with students Institutional research representative Day 142 
(improvement team members) 

Assess results and document findings | Improvement team lead (improvement team | Day 165 
members, institutional research staff) 

Share results and gather input Improvement team lead (advisers, faculty, Day 172 
students) 

Decide whether and how to scale, Improvement team lead (improvement team | Day 180 

refine, or abandon the change members, other key stakeholders) 

Identify new issues and questions for | Improvement team lead (improvement team | Day 180 


improvement efforts 


members) 
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sions. Program documentation is collected to document the auditing and refinement 
process. Faculty in the relevant programs are sent a one-question survey to assess 
satisfaction. After collecting and analyzing the program documentation and faculty 
data, you decide that changes will be rolled out to advisers and students for four 
programs. Advisers are assessed on their knowledge of the changes post-training, 
and students are assessed through a few focus groups for each of the programs. 


Study 

It is now time to assess the results, share them to gather input from a broad variety 
of stakeholders, and record the findings. You find that the program staff conduct 
audits for three of their certificate programs in each of the two fields. Of the six 
programs audited, staff decide to make changes to streamline course requirements 
for five. Faculty survey results suggest that faculty are comfortable with the changes 
in four of the five programs. However, for one of the streamlined programs, a large 
proportion of faculty members report concerns about one of the courses that was 
cut from the associate’s degree program. As a result, you decide to push forward 
with testing changes for those four programs that had faculty support. Data from ad- 
viser assessments suggest that the training was effective in providing the informa- 
tion that advisers needed, but student focus group findings suggest that more might 
be done to improve student awareness of the changes. You share these findings 
broadly with advisers, faculty, and students to hear their interpretations of the find- 
ings, solicit input to inform decisionmaking on next steps, and familiarize them with 
evidence supporting the mandatory advising requirements to build buy-in. 


Act 

Now that you have assessed the evidence, it is time to make some decisions. First, 
will you scale the improvement to additional programs or abandon the change and 
test out a new one? The institution abandoned changes for two of the six initial 
programs midstream because of evidence suggesting that streamlining these 
programs to a greater degree was not something practitioners supported. For the 
other four programs in which changes to course requirements were made to stream- 
line programs, evidence suggests that more might be done to get the word out to 
students, so you decide to test out these informational changes in a PDSA cycle. You 
would also like to expand this process to other fields and programs through addi- 
tional PDSA cycles, repeating Step 4 until the change is successfully scaled. Finally, 
you and your team document other issues and questions related to the stackable 
credential pipeline that might be addressed through future PDSA cycles. 


Your team continues to engage in PDSA cycles, testing and scaling improvements 
to streamline program course requirements across stackable credential programs. 
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After 18 months, you fail to achieve your aim of 60 percent of all credit hours for certifi- 
cate programs counting toward the degree, so you extend the timeline for an additional 
12 months and continue to make changes until the aim is met. 


Final Considerations for Using 
the Four-Step Process 


Carrying out this four-step continuous improvement process can be valuable in helping 


postsecondary institutions improve policy and practice in several ways. First, it provides 
organizations with an approach to dig deeper into issues they face and encourages 
them to focus on specific, “bite-sized” problems of practice that can be tackled ina 
relatively short period of time. Second, it provides a more structured process for piloting 
different improvements that an organization might want to make and assessing wheth- 
er these improvements were in fact valuable. It encourages organizations to refocus 
data use for the purpose of informing practice rather than just using data for account- 
ability and grant reporting. Third, the practitioner-driven orientation of the approach 

(as opposed to the top-down or external nature of many evaluations) can facilitate the 
organization’s ability to pinpoint the critical issues, design realistic improvements, fully 
assess the findings from multiple perspectives, and ensure buy-in as improvements are 
scaled. Finally, the four-step approach is flexible and can be used to drive improvement 
on a wide variety of educational policies and programs; once learned, the process can 
be applied broadly throughout the institution. 


However, continuous improvement is not without its challenges. The process takes 
significant time and effort, requiring a strong commitment to following through with 

the full process and buy-in at all levels. Table 15 highlights some examples of barriers 
that RAND researchers have encountered in working with postsecondary institutions 

to implement continuous improvement. Some of the most commonly reported barriers 
include finding sufficient time to meet, concerns about burdening people with data 
collection, a lack of buy-in from certain departments, and a lack of authority to drive 
change. The table provides some examples of strategies that can be used to overcome 
these barriers. For example, providing the continuous improvement team with dedicat- 
ed time to oversee the work or providing a stipend might help ensure that individuals 
can find time to meet, and using existing data sources can help address concerns 
about the burden on participants. Departments that lack buy-in should be brought into 
the process, either by adding one or more members from that department to the team 
overseeing the improvement work or by soliciting input from the department as the 
team formulates the problem of practice and then determines which changes will be 
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Table 15. Barriers to Continuous Improvement and Strategies to Overcome Them 


Common Barriers 


Examples of Strategies to Address the Barriers 


Lack of knowledge on how to 
carry out the process 


Provide training and resources to support internal staff, hire outside 
consultants 


Lack of support from 
leadership 


Provide examples of the value of continuous improvement, collect and docu- 
ment evidence, engage leadership frequently, include leadership on the core 
continuous improvement team 


Lack of buy-in from other 
groups that are key to 
improvement efforts 


Ensure that the process is transparent, invite participation in the core 
improvement team, focus on problems of practice and improvements that are 
meaningful to the group, collect and document evidence 


Siloes that limit participation 
of key groups 


Ensure a diverse core improvement team that represents key groups, engage 
stakeholders through data collection 


Too little time to meet 
and carry out the work, 
distractions 


Identify resources and carve out explicit time for the work, include individuals 
who value the process, make the process high-visibility, set clear intermediate 
milestones, hold individuals accountable for the results 


Time and burden associated 
with data collection 


Use existing sources wherever possible, keep data collection focused on key 
measures, communicate the value of data collection 


Limited expertise with data 
collection 


Include staff with data expertise on the core team, provide training and 
resources to support internal staff, hire outside consultants 


Institutional policies and 
procedures that delay data 
collection or change efforts 


Build buy-in with leadership, explore ways to increase the flexibility of policies 
and procedures that are hindering quick testing and change, collect and 
document evidence 


Lack of authority to take 
action based on evidence 


Include individuals on the team with authority to make change, focus on issues 
that the team has authority to change, build buy-in with leadership, explore 
ways to increase the flexibility of policies and procedures that are hindering 
quick testing and change, collect and document evidence 


Cultural resistance to change 


Identify individuals for the core team who are willing to challenge beliefs, build 
buy-in with leadership and key stakeholders, collect and document evidence, 
engage key stakeholders throughout the process 


tested. It is also important to ensure that the process is transparent and that the 
evidence being used to justify decisions is clearly documented to build buy-in 
and convince stakeholders that changes should be made. Once postsecondary 
institutions have overcome their barriers to continuous improvement, they can 
more easily incorporate continuous improvement as a central part of the process 
of rolling out important new initiatives or programs. 
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Supplementary Worksheet: Fishbone Diagram 


Problem 


N t 
a > 
oO ro) 
oD) 5) 
Y oO 
~ ~ 
© © 
SI O 
= on] 
ra ra 
: g 
o o 
SD CI 
rf rf 
O O 


Note: Fishbone diagrams can be adapted to include as many categories and bones as needed, and participants will need to 


draw in bones to capture secondary drivers. 
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Supplementary Worksheet: Model for Improvement 


Aim: What are you trying to 
accomplish? 


Measures: How will you 
know that a change is an 
improvement? 


Changes: What changes will 
you make that will result in 
improvement? 


MAKING IMPROVEMENTS TO STACKABLE CREDENTIAL PIPELINES 


Supplementary Worksheet: Plan-Do-Study-Act (PDSA) Cycle Data Collection and Analysis Plan 


Question Prediction Data Source 
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Supplementary Worksheet: Activities, Roles and Responsibilities, and Timeline 


Activity Lead (Other Participants) Deadline 


